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Storage System (and the Kernel) 
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Who am I? 



• Soy Christian Kendi 

• I do ... 

- IT-Security Consultant 

- (Kernel)-Developer 

- Penetration tester 

- Exploit coder 

- CEO & Founder of Iron Software 
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What is this->talk about? 

ZFS (Zetabyte File System) 
Open Solaris Gate (Kernel) 
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What is this->not about? 



• Further explanation on how file systems 
work in general 

• Deeper insight into the design and 
development of ZFS (raidz, allocator, etc.) 

• Rootkits (well ©), we are not too far away 
from a rootkit 
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What is ZFS? 

• Revolutionary Open Source Storage 
System 

• 128-bit file system 

• Capable of storing 16 EiB (1,024 
Pebibytes) 

• Transparent Compression, Encryption, 
etc... 

• Ported to multiple Operating Systems 
(Mac OS X, BSD, Linux) 

/HUrTSOttSMdte Christian Kendi 



ZFS features 



• Storage pools 

• Snapshots 

• (Incremental) Backups between Snapshots 

• Variable block-size up to 128-kilobyte 

• On-the-fly compression (LZJB, gzip[1-9]) 

• 256-bit block checksums (fletcher2/4 or 
SHA-256) 

• Self Healing (On-the-fly Error Correction) 

• Open Source (yes its a feature ;) 

IROrkatimiK* Christian Kendi 
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ZPL (ZFS POSIX Layer) 

ZVOL (ZFS Emulated Volume) 

DMU (Data Management Unit) 

DSL (Dataset and Snapshot 
Layer) 

ZAP (ZFS Attribute Processor) 

ZIL (ZFS Intent Log) 

ARC (Adaptive Replacement 
Cache) 

Pool Configuration (SPA) 

7 

ZIO (ZFS I/O Pipeline) 



DEMO 



Create a pool, filesystem and 
work with snapshots 
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Security aspects about ZFS 



• Offline honey pot analysis 

• Backup's of Mission Critical Systems 

• Embedded Antivirus support for blocking 
infected files 

• Revert a hacked host back to installation 
state within seconds 

• Forensics by differential FS analysis 

• ACLs 
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Storage Security Concerns 



• The most valuable information is stored in 
databases and storage Systems 

• Having access to the company's storage 
equals having the company 

• More to come later. . . 
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ZFS enhancing, how? 



• A file system is always kernel based 

• Open Solaris ON NV (Gate) Source Code 

• Building a kernel module 

• Hooking internal ZFS functions 

• Provide a separate FS-Layer for the 
"enhances" 
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ZFS enhancing, helpers? 



• kmdb is incredible! 

• dtrace & truss 

modules::list "struct modctl" mod_next | ::print "struct modctl" 
{ 

mod_next = 0xfec479e0 
mod_prev = 0xda0564c8 
modjd = 

mod_mp = 0xfec42d90 
mod_inprogress_thread = 
modjnodinfo = 
modjinkage = 

mod_filename = 0xfec42d68 7platform/i86pc/kernel//unix" 
mod modname = 0xfec42d80 "unix" 
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ZFS enhancing, how? #2 



• movl $add, %eax; jmp *%eax 

• Assembly code injection 









0xf9e7635c: 


nou 1 


SOxf alZZf 50 ,xeax <zf s * zf s_nkd ir> 


0xf9e7636i: 


jnp 


**eax 


0xf9e76363= 


inb 


Cxdx) 


0xf9e76364: 


dec 1 




0xf9e76365: 


nou 1 


OxB Cxebp ) , y.eAx 


0xf9e76368: 


nou 1 


OxlO(>:eax) ,>iesi 


0xf9e7636b= 


nou 1 


C^es i ) j ^ebx 
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ZFS enhancing, how? #2 



• But of course we don't want to rewrite the 
entire ZFS code. 



[Q]> *or ig_zf s_dir look : =dis 






0xf9e6821c: 


mou 1 


§0xf a2813aG , xeax <zf s 1 zf s_d ir look> 


0xf9e6822i: 


jnp 


*xeax 


0xf9e68223= 


addb 


y.A 1 j (xeax ) 


0xf9e68225: 


addb 


y.a. 1 j (^eax ) 


0xf9e68227= 


addb 


y.cl , (xedi ) 


0xf9e68229: 


mou 1 


§0xf 008502,^051 


Oxf 9e6822e : 


testb 


xbh , 0x83000001 Cxebp ) 


0xf9e68234: 


c lc 




0xf9e68235: 


jne 


+0xlb <0xf9e68Z53> 


0xf9e68238: 


nousb 1 


Oxl(^edx) ,^eax 


Pxf 9e6823c : 


test 1 


xeax j xeax 



• First bytes are restored when executing from 
the orig_handler within the hook. 
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ZFS enhancing, what? 



dtrace and truss are your friends 
Find the desired functions 



-> getdents64 
-> getf 

-> set_active_f d 
<- set_active_f d 
<- getf 
-> fop_rwLock 
-> fs_rwLock 
<- fs_rwLock 
<- fop_rwLock 
-> fop_readdir 
-> crgetmapped 
-=:- crgetmapped 
-> zfs_readdir 
-> rrw_enter 
-> rrw_enter_read 
<- rrw_enter_read 
renter 




-> zap_cursor_init_serialized 
<- zap_cursor_init_serialized 

m l Log 

-> kmem_cache_a L Loc 
<- kmem_cache_a L Loc 
-=:- kmem_al loc 
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ioetL(3, 


ZF3_I0C_0B33ET_5TAT5 , 0X08O450C0) 







brk(0x030B4000) 
ioctL(3, ZFS.IOC. 


_POOL_STATS, 0x03045020) 








ioctL(3, 
HrtJcTT(3", 


ZFS 


mn 


pnm rft ppnpq RvHOMfiHRC^ 







ZFS. 


,IOC_DATASET_LIST_NEXT , 0X080460E0Y 






ioctl(3, 


ZFS. 
_ZFS_ 


.IOC. 
.IOC. 


_DATASET_LIST_NEXT , 0X08O460EO) 
_DATASET_LIST_NEXT , 0x080460Efi). 




^) 


ioctL(3, 


ZFS. 


.IOC. 


.DA 1 Abb 1 1 _NbX 1 , UXU80460E0) 


Err#3 ESRCH 


ioctl(3, 


ZFS. 


.IOC. 


.OBJSET.STATS , 0X080450C0) 







ioctL(3, 


ZFS. 


.IOC. 


.POOL_STATS, 0x03045020) 







ioctL(3, 


ZFS. 


.IOC. 


.P00L_GET_PR0PS, 0x08046080) 







ioctl(3, 


ZFS. 


.IOC. 


_DATASET_LIST_NEXT, 0X08O460E0) 







ioctL(3, 


ZFS. 


.IOC. 


_DATASET_LIST_NEXT , 0X08O460E0) 







ioctL(3, 


ZFS. 


.IOC. 


_DATASET_LIST_NEXT , 0X080460E0) 







ioctL(3, 


ZFS. 


.IOC. 


.OBJSET.STATS , 0X08044020) 







ioctL(3, 


ZFS. 


.IOC. 


_DATASET_LIST_NEXT , 0X08O460EO) 







ioctl(3, 


ZFS. 


.IOC. 


.OBJSET.STATS , 0x08044020) 







ioctl(3, 


ZFS. 


.IOC. 


_DATASET_LIST_NEXT, 0x08045040) 







ioctL(3, 


ZFS. 


.IOC. 


_OBOSET_STATS , 0X08042F80) 


Err#12 ENOMEM 


ioctl(3, 


ZFS. 


.IOC. 


.OBJSET.STATS, 0X08042F80) 
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ZFS enhancing, syscalls? 

• Okay, admitted. Old but nice. Why? 

• Crypto Gate ;) 

• Just to be flexible 

switch ( zc->zc_crypto . zic_cmd ) ■[ 
case ZF S_IOC_CR Y P T 0_L AD_K S Y_£ PA : 

error spa_crypto_key_load ( spa x £zc->zc_crypto ) } 

break; 

case 2 F £_1 C_CR Y P T 0_UWL AD_K E Y_S P A : 

error ■ spa_crypto_key_unload ( spa ) ; 
break; 

case 2 F £_3 C_CR Y P T 0_C B AH C E_K E Y_S P A : 

error = spa_crypt_key_change ( spa , izc->zc_crypto ) ; 
break; 
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ZFS enhancing, functions? 

• All userland <-> kernel communication is 
in zfs_ioctl.c 

• zfs_ioc_pool_configs() will deliver all 
available pools 

- Or not 

• Solaris handles dynamic data with nvlists 

• Dynamic means DYNAMIC. 
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ZFS enhancing, 



nvlists 



nvlist_t 

nvl_versioo 
nvl_nvflag 
nvl_priv 
nvl_f lag 
nvl_pad 



I 

■I 

I 

-+-+ 

I I 
I 

-+ I 



nvpiriv_t 



1 

■+- nvp_list 
nvp_last 
nvp_curr 
nvp_nva 
nvp_3tat 



- ■ 

V 



i_nvp_t 



nvi_next 
nviprev (HULL) 

nvp ( nvpair_t ) 

- nvp_size 

- nvp_name_sz 

- nvp_value_elem 

- nvp_type 

- data . . . 



last i_nvp in list 
> 

+ + 

+ nv_alloc_t [ 
| [ 

■-> | nva_ops 
| ova_arg 
+ + 

.+ + + 

| + — >\ invpt | + — > 

I I I— - 1 I 

■+ — ^ nvinext -+ — + 

< + nviprev 

I 

nvp { nvpair_t ) 

- nvp_size 

- nvp_name_sz 

- nvp_value_elem | 

- nvptype 

- data . . . 

.+ 4- + 
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ZFS enha 



/* VOPS HERE */ 
dsym(int , 
dsym(int, 
dsym(int , 
dsym(int, 
dsym(int , 
dsym(int, 
dsym(void , 
dsym(int, 

/* ZFS INTERNALS V 
dsym(int , 
dsym(int , 
dsym(int , 
dsym(int, 



/* ZFS IMPORTS - 
isym(int , 
isym(nvlist_t *, 
isym(int , 
isym(int , 



ONLY 



zf s_dirlook, 
zf s_mkdir, 
zf s_zaccess, 
zf s_read, 
zf s_write, 

zf s_ i oc_poo I _get_h i story , 

zf s_log_history, 

zf s_ i oc_poo I _conf i gs , 



zf s_ i oc_poo I _stats , 
zf s_ i oc_dataset_ I i st_next , 
ds I _d i r_ i s_pr i vate , 
dataset_namecheck , 

*/ 

spa_get_stats , 
spa_al l_conf igs, 
put_nvlist, 
dataset_narne_h i dden , 



/* VOPS IMPORTS - 


- ONLY -*/ 


isym(int, 


zf s_open, 


isym(int , 


zf s_close, 


isym(int, 


zf s_ioctl , 


isym(int , 


zf s_access, 


isym(int, 


zf s_ lookup, 


isym(int , 


zf s_create, 



isym(int, zfs_remove, 

isym(int, zfs_rmdir, 

isym(int, zfs_readdir, 

isym(>.nt, zfs_fsync, 

isym(int, zfs_ inactive, 

isym(int, zfs_getattr, 

isym(int, zfs_setattr, 

isym(int, zf s_renarne, 

isym(int, zfs_syrnlink, 

isym(int, zfs_readlink, 

isym(int, zfs_link, 

isym(int, zfs_seek, 

isym(int, zfs_fid, 

isym(int, zf s_pathconf , 

isym(int, zf s_getsecat.tr, 

isym(int, zf s_setsecattr, 



/Rnnsotbwuf! 



znode_t *dzp, char *name, vnode_t **vpp, flags, *def Ig, pathnarne_t *rpnp); 
vnode_t *dvp, char *dirnarne, vattr_t *vap, vnode_t **vpp, cred_t *cr); 
znode_t *zp, int mode, int. flags, boolean_t skipaclchk, cred_t *cr); 
vnode_t *vp, uio_t *uio, int i of lag, cred_t *cr, ca I ler_context_t *ct); 
vnode_t *vp, uio_t *uio, i of lag, cred_t *cr, ca I ler_context_t *ct); 
zfs_cmd_t *zc); 
zfs_cmd_t *zc); 
zfs_cmd_t *zc); 



zfs_cmd_t *zc); 
zfs_cmd_t *zc); 
dsl_dir_t *dd); 

const char *path, namecheck_err_t *why, char *what); 



const char *name, nvlist_t **config, char *altroot, size_t buf len); 
uint64_t *generation); 
zfs_cmd_t *zc, nvlist_t *nvl); 
const char *name); 



vnode_t **vpp, int flag, cred_t *cr, ca I ler_context_t *ct); 

vnode_t *vp, int flag, int count, offset_t offset, cred_t *cr, cal ler_context_t *ct); 

vnode_t *vp, int corn, int.pt.r_t data, int flag, cred_t *cred, *rvalp, cal ler_context_t *ct); 

vnode_t *vp, int. mode, int. flag, cred_t *cr, ca I ler_context_t *ct); 

vnode_t *dvp, char *nm, vnode_t **vpp, pathname *pnp, int. flags, 

vnode_t *rdir, cred_t *cr, cal ler_context_t *ct, int *direntf lags, pathname_t *realpnp); 

vnode_t *dvp, char *name, vattr_t *vap, vcexcLt excl, mode, vnode_t **vpp, 

cred_t *cr, int flag, cal ler_context_t *ct, vsecattr_t *vsecp); 

vnode_t *dvp, char *name, cred_t *cr, ca I ler_context_t *ct, int flags); 

vnode_t *dvp, char *name, vnode_t *cwd, cred_t *cr, cal ler_context_t *ct, int flags); 

vnode_t *vp, uio_t *uio, cred_t *cr, int *eofp, ca I ler_context_t *ct, int flags); 

vnode_t *vp, int syncf lag, cred_t *cr, ca I ler_context_t *ct); 

vnode_t *vp, cred_t *cr, cal ler_context_t *ct); 

vnode_t *vp, vattr_t *vap, int flags, cred_t *cr, ca I ler_context_t *ct); 
vnode_t *vp, vattr_t *vap, int flags, cred_t *cr, ca I ler_context_t *ct); 

vnode_t *sdvp, char *snm, vnode_t *tdvp, char *tnm, cred_t *cr, cal ler_context_t *ct, int flags); 
vnode_t *dvp, char *name, vattr_t *vap, char * I ink , cred_t *cr, cal ler_context_t *ct, int. flags); 
vnode_t *vp, uio_t *uio, cred_t *cr, ca I ler_context_t *ct); 

vnode_t *tdvp, vnode_t *svp, char *name, cred_t *cr, cal ler_context_t *ct, int flags); 
vnode_t *vp, offset_t ooff , offset_t *noffp, ca I ler_context_t *ct); 
vnode_t *vp, fid_t *fidp, cal ler_context_t *ct); 

vnode_t *vp, int cmd, ulong_t *valp, cred_t *cr, ca I ler_context_t *ct); 
vnode_t *vp, vsecattr_t *vsecp, int flag, cred_t *cr, cal ler_context_t *ct); 
vnode_t *vp, vsecattr_t *vsecp, int flag, cred_t *cr, cal ler_context_t *ct); 
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ZFS, The Hackers point of View 

• Hide "something" 

• Anti-forensics against unloading the 
module 

• + Hide data in a way that offline analysis is 
hard 

• Yes, Crypto is a solution, but.... 

- the key must be stored _somewhere_ 
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ZFS, The Hackers point of View #2 

• Some ideas... 

- a private storage pool 

- mirror the companys pool over the internet. 
(iSCSI, zfs send) 
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ZFS, The Hackers point of View #3 

• Interesting ioctl's 

ZFS_IOC_SEND 

ZFS_IOC_RECV 

ZFS_IOC_SNAPSHOT 

ZFS_IOC_POOL_STATS 

ZFS_IOC_POOL_GET_PROPS 

ZFS_IOC_POOL_CONFIGS 

ZFS_IOC_SNAPSHOT_LIST_NEXT 

ZFS IOC DATASET LIST NEXT 
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ZFS enhancing, hide something? 



• It's all there by it-self, ".zfs" is invisible 

• Analysis and code reading/auditing 
revealed interesting stuff 

4 include 'zfs namecheck.h 
int 

dataset name hidden (const char *name) 
^ifjSrcTiit^ 

return (0); 

} 
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ZFS enhancing, hide something? 

#2 

• .zfs is a VFS (Virtual File System) layer by 
itself 

• With ZFS we don't just hide directories or 
files, we hide entire file systems or storage 
pools 

- Each hidden FS/pool has its own VFS entry 

- VFS controls all FS specific operations 

VOPNAMEJ-OOKUP, { .vopjookup = ksh_root_lookup }, 

- > Have different ZFS revisions in a single kernel 

- ZFS Crypto Gate 
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ZFS enhancing, hide someth 

#3 

in| 

zf s_dir Look(znode_t *dzp, char *name, vnode_t **vpp, int flags, 
int *deflg, pathname_t *rpnp) 

{ 

zfs_dirlock_t *dl; 
znode_t *zp; 
int error = 0; 

(name[0] == 9 I I (name[0] == ' . ' £& name[l] == 0)) { 
*vpp = ZTOV(dzp); 
VN_HOLD(*vpp); 

} else if (name[0] == ' . ' £& name[l] == '.'££< name [2] == 0) { 
zfsvfs_t *zfsvfs = dzp->z_zf svf s; 
/* 

* If we are a snapshot mounted under .zfs, return 

* the vp for the snapshot directory. 
*/ 

f (dzp->z_phys->zp_parent == dzp->z_id £& 
zf svf s->z_parent != zfsvfs) { 

error = zfsctl_root_lookup(zfsvfs->z_parent->z_ctLdir, 
"snapshot", vpp, NULL, 0, NULL, kcred, 
NULL , NULL, NULL); 
1 (error); 

} 

r w_enter (£jdzp->z_par ent_ L ock , RW_RE ADER ) ; 
error = zf s_zget(zf svf s, dzp->z_phys->zp_parent , &zp); 
(error ==0) 

*vpp = ZTOV(zp); 

rtt\Mizp->z_parent_ lock) ; 




F (zfs_has_ctldir(dzp) strcmp(name, ZF5_CTLDIR_NAME) == 0) 
*vpp = zfsctl_root(dzp); 
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ZFS enhancing, Anti-forensics 

• ZFS binary and kernel module contain checks for invalid datasets, 
i.e. internal datasets 

• Built-in support for hiding Storage Pools and ZFSs across Systems. 

Apr 9 13:06:16 opensolaris-vm winnipu: [ID 181094 kern . warning] 
WARNING: hook_zf s_ioc_dataset_list_next ( ) : zc_name: rpool/$MOS cookie: 
133e8aad 

Apr 9 13:06:17 opensolaris-vm winnipu: [ID 181094 

kern . warning] WARNING: hook zfs ioc dataset list_next() : 

zc name: rpool/$ORIGIN cookie: 13763f21 

• Module independent, Oday? © 

• zfs send independent 

• Snapshot resistant 

• Pools and ZFS's wont show up even if module is not loaded 

• Takes advanced personnel to find the pool 
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ZFS enhancing, Anti-forensics #2 



• Patch zfs binary to allow ,$' 

• Hook dataset_namecheck() 

-Allow "all" characters to special PIDs 

• LD PRELOAD recompiled libzfs.so.1 with 
new zfs binary 

• list, create, snapshot, send/recv, etc... 
considered internal datasets 
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Anti-debugging? 



• Because, the code is all mine. 

• Symbol relocation is done is in the Elf header 

• The Module pointer holds mp->symtbl 

• sp = (Sym *)(mp->symtbl + i * mp->symhdr- 
>sh_entsize); & sp->st_value = Ox???????? 

is your friend 

• kobj_sync() refresh's the module symtab 

• Have fun debugging Oxfe?????? and 
m0e3asd 
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DEMO 



Let's make some magic 
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Poopool 



What is "poopool"? 



$ ./new_zfs.sh list poopool 
sending request for PID 806... done! 
NAME USED AVAIL REFER MOUNTPOINT 

poopool 40. 3M 123M 40. 1M none 

$ zpool status poopool 
pool: poopool 

state: ONLINE 

scrub: none requested 
conf ig : 



NAME STATE 
poopool ONLINE 
/root/poopool . raw ONLINE 



READ WRITE CKSUM 






errors: No known data errors 
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Hidden subpools with "$" 



• poopool/$bleh 

• Won't automount (nice) 

NAME USED AVAIL REFER MOUNTPOINT 

poopool/$bleh 18K 123M 18K /system/ . zfs/asdf 

• Everything about ZFS can be logged 



Jan 4 18:24 


:21 


openso 1 ar i s- um 


ksh. 


_zf s : 


WARM IMG : 


hook. 


_ioctl() 


: 2FS_I0C_ 


POOL_GET_ 


PROPS 




















Jan 4 18=24 


:21 


openso lar is- urn 


ksh. 


_zf s : 


WARM IMG : 


hook 


_zf s_Iog. 


_li i story C ) 


: log que 


ry : poopool 




















Jan 4 18=24 


:21 


openso lar is-un 


ksh 


_zfs : 


WARN IMG : 


hook. 


_zf s_log. 


_h istory C ) 


: zc_name 


: poopool 




















Jan 4 18=24 


■Zl 


openso lar is-vm 


ksh 


_zfs = 


WARM IMG : 


hook. 


_zf s_log. 


_h istory C ) 


: denying 


history log 


on 


poo 1 : poopoo 1 
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Let's sum it up 



• Kernel hacking 

• Some ZFS internals 

• VFS Layers 

• Dynamic Symbol Relocation 
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Outlook 



• Hot-patching mission critical systems 

• Implementing new (desired) features into a 
running system 

• Adapting a second protection layer 

• ZFS Crypto gate in code review (still) 
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Questions? 



34 

//TO/TSOftWare Christian Kendi 



Thanks for listeni 

Have fun! 
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